Kiefer Wolfowitz Algorithm is Asymptotically Optimal for a Class of Non-Stationary Bandit Problems

We consider the problem of designing an allocation rule or an “online learning algorithm” for a class of bandit problems in which the set of control actions available at each time t is a convex, compact subset of R. Upon choosing an action x at time t, the algorithm obtains a noisy value of the unknown and time-varying function ft evaluated at x. The “regret” of an algorithm is the gap between its expected reward, and the reward earned by a strategy which has the knowledge of the function ft at each time t and hence chooses the action xt that maximizes ft. For this non-stationary bandit problem set-up, we propose two variants of the Kiefer Wolfowitz (KW) algorithm i) KW with fixed step-size β, and ii) KW with sliding window of length L. We show that if the number of times that the function ft varies during time t is o(T ), and if the learning rates of the proposed algorithms are chosen “optimally”, then the regret of the proposed algorithms is o(T ). I. KIEFER WOLFOWITZ ALGORITHM We begin by describing the KW algorithm for the case when the function f to be optimized is fixed, and the maximizer is denoted θ(f) ∈ D. The vanilla version of the KW algorithm maintains, at each time-step n an estimate of the function minimizer, denoted as Xn. It then makes an estimate of the derivative of the unknown function f by sampling the function values at points Xn+ cn and Xn− cn. Let Y + n , Y − n be the noisy values of the function at Xn+cn and Xn−cn respectively. Denote by Yn the estimated value of the derivative of function f at Xn. We then have that,